Framework for combining content intelligence modules

ABSTRACT

A method for analyzing media assets such as video and audio files. The method includes providing access to all the frames of a digital media asset. The method includes, with a microprocessor, running a raw analyzer modules to analyze the asset frames to produce sets of raw analyzer result data that are stored in a data cache in a file associated with the asset. The sets of raw analyzer results are linked to the raw analyzer modules with unique identifiers. The digital media asset is played for the raw analyzer modules, which concurrently analyze the temporally-related frames. The raw analyzer results are stored as data tracks that include metadata for the asset such as immutable parameters including histograms. The method includes using a feature algorithm module to generate an analysis result, such as face identification, for the digital media asset based on the raw analyzer results accessed by the identifiers.

BACKGROUND

1. Field of Description

The present disclosure relates, in general, to computer-implementedmethods for running content intelligence algorithms or software moduleson digital media assets such as video images, and, more particularly, toimproved methods and systems for combining content intelligence modulesand output/results of such content intelligence modules for moreeffective use by applications.

2. Relevant Background

Recently, there have been many advances in software algorithms ormodules that are useful in analyzing digital media to provideinformation about the media. For example, a digital asset, such as aframe of a video source or a digital image, may be analyzed with acomputer application to automatically determine whether the assetincludes a human face. If so, another application or module may act todetermine whether the face belongs to a specific person, which may havenumerous uses such as searching for images of a particular person inlarge asset sets such as video or image databases accessible via theInternet or to determine from surveillance cameras whether a suspectedcriminal has been in proximity of a particular camera location. Otheralgorithms or software modules may be used to provide other informationsuch as facial expression, activity in a frame or image, a shot in avideo, a brightness level of an image, and/or other specific informationfor a media asset. This collection of algorithms or modules may belabeled content intelligence modules or algorithms.

In general, each content intelligence algorithm is created to perform aparticular task or function with relation to a media asset. Each contentintelligence algorithm such as a face identifier algorithm for use withstill images may output a set of result or output data. Unfortunately,most content intelligence algorithms do not return data that can be useddirectly as a feature or the like. Instead, the content intelligencedata or results have to be post-processed to be useful, and often thepost-processing further requires that the data from differing algorithmsbe combined to be used, e.g., brightness levels on their own may not beuseful, activity identified in an image may not be useful withoutfurther data, and so on. Another reason that the content intelligenceresults often have to be post-processed and combined is that eachcontent intelligence algorithm provides its output in the context oftheir specific environment. It is left up to another application oranother content intelligence module to determine that context toproperly use the results, which may make it difficult to properlycombine or build upon the results of another content intelligencealgorithm.

SUMMARY

Developing content intelligence software modules to provide desiredartificial intelligence and analysis of media data is a challengingtask. Combining a number of content intelligence algorithms in arelatively hardwired or orderly manner has not been adequately achievedand has presented numerous obstacles, which are heightened andemphasized when any change is later performed or implemented.

The following description provides methods and systems that allow quickand effective combination of content intelligence (CI) algorithms ormodules in an orderly way. This combination allows the CI modules tosupport each other to use the functionality and results/outputs of otherCI modules to generate collaborative and/or improved results (e.g.,post-processing of CI module results or data is enhanced andsimplified). The CI framework or toolkit may be thought of as a softwareframework that facilitates the combination of various CI modules oralgorithms to form features (outputs or results of combined functioningof two or more CI modules) that can then be used by one or more mediaanalysis/processing applications. The CI framework may be adapted toprovide a generic interface that can be used by such applications toobtain the results of the CI modules and/or features.

More particularly, a computer-implemented method is provided foranalyzing a media asset such as a video or audio file. The methodincludes providing sequential access to a plurality of portions of adigital media asset (with “sequential” being intended to convey that theportions are generally played in their time-related sequence or orderbut a raw analyzer may request more than one and/or review the portionsout of order). The method also includes, with a microprocessor, runninga plurality of raw analyzer modules (RAs) to analyze the portions of thedigital media asset to produce sets of raw analyzer result data. Themethod also may include storing in a data cache or data store the setsof raw analyzer result data in a file that is associated with thedigital media asset (e.g., typically one data cache file per asset).Each of the sets of raw analyzer result data may be linked to aparticular one of the raw analyzer module such as with a single uniqueidentifier (UID).

In some embodiments, the digital media asset comprises an audio or videofile that is played in its entirety for the RAs, which may concurrently(at least partially) analyze the temporally-related frames of the file(which may include timestamps indicating their time relationships). Inan implementation of the method, the RA results are stored as datatracks that provide metadata for the frames of the video/audio file thatare extracted by an associated one of the RAs. The results or metadatamay be associated with the proper frame using the timestamps found inthe video/audio file. The extracted metadata may include substantiallyimmutable properties or parameters for the content of the video/audiofile (e.g., histograms or the like).

The method may also include running, with a microprocessor, a featurealgorithm module (FA) to generate a feature or analysis result (such asshot/scene determination or face/logo/object identification) for thedigital media asset based on or by using one or more of the sets of rawanalyzer result data from the data cache. For example, the RA resultdata may be accessed by the FA by providing or using a UID associatedwith a particular one of the RAs, and, in some embodiments, theparticular RA and the calling FA are provided within a plugin run by amicroprocessor (e.g., as part of a content intelligence (CI) frameworkor toolkit). In some cases, the method may include a microprocessorrunning an additional/second FA that creates an additional/secondanalysis result for the asset based on the analysis results of the firstFA and/or one or more of the RA result sets or data tracks (which may bethe same or different than the RA result sets used by the first FA).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates in functional block form a computer systemimplementing media processing or analysis of media assets using acontent intelligence toolkit or framework including use of plugins withraw analyzer (RA) and feature algorithm (FA) modules;

FIG. 2 illustrates a flow diagram of a method of running a set of RA andFA modules to process media data or a media asset (e.g., a temporalmedia asset such as a video file);

FIG. 3 illustrates a CI environment in functional block form that may beimplemented as software modules within one or more computer and/ormemory devices;

FIG. 4 is a graph or schematic illustration of data dependencies betweenfeature algorithm modules and generating raw analyzer modules; and

FIG. 5 is a schematic illustration or functional block drawing ofoperation of a content intelligence framework to analyze a media asset.

DETAILED DESCRIPTION

The following description describes the use of software (and/orhardware) implementations to provide a content intelligence toolkit thatprovides a software framework that combines a number of contentintelligence (CI) algorithms or modules (e.g., raw analyzers or RAs) soas to form features or feature algorithms or modules (FAs), which can,in turn, be used (or their results/outputs used) by mediaanalysis/processing applications or other applications. In brief, the CIframework described provides a number of RAs, and, during operation, amedia asset or media data such as a video file may be processed by theRAs to generate a set of content data that is stored in memory. Forexample, a video file may be played in its entirety and information maybe extracted by each RA during this playing of the file to create datacache files for the media asset. Such information may be stored as datatracks in the data cache file of the asset with each data track timestamped and linked to the producing RA by a unique identifier. A set ofFAs may then take the input of the RAs (or select ones of the RAs) andproduce a feature result that may be stored as a tag that is associatedwith the media asset and with the producing FA (such as by anotherunique identifier). An application such as Adobe's PhotoShop or othermedia analysis/processing application may access the tags or FA outputsto further analyze or process data (metadata) associated with the mediaasset.

In this description, it may be useful to provide definitions for anumber of several terms/acronyms. For example, content intelligence orCI is a relatively general term that is used to describe anysoftware/hardware tool or module that is used to analyze or make use ofthe analysis of content of a media asset such as temporal media (e.g.,audio and/or video files of digital data) or still media (e.g., imagesor still image files). A content intelligence toolkit (CIT) or CIframework is a collection of shared software components or modules thatbring basic CI functionality to implementers and users of CI features toprocess media assets.

The term unique identifiers or UIDs is used to refer to certain objectsor classes of objects. For example, UIDs may be used to identify a rawanalyzer data track for a media asset or a feature algorithm tag. Inthis regard, a raw analyzer (RA) is a software module (which may beprovided as part of a plugin with an FA module) that functions when runby a microprocessor to extract properties or information (e.g.,immutable properties) from media assets such as histogram data fromvideo or audio frames. Output data of an RA may be stored in a datacache as a data cache file for each asset, with the RA and its outputdata being linked (such as by use of the same UID or the like). Afeature algorithm or FA is a software module that can be run by amicroprocessor to turn or modify data from the data cache file (e.g., RAdata or data tracks generated from an RA module) into a list of tagswith this FA output being linked to the FA (such as by use of single UIDfor the tag (or type of tag) and the FA module). A “tag” in this senseis a product or output of a feature algorithm or FA module and may beused to describe a segment of footage or a media asset where a certaindescription applies (e.g., “Scene number 1 ranges from 0.0 to 3.12seconds” or the like), and the tag may be stored in the data cache or ina separate data store such in XMP. The data cache may be a data storageor memory that is used to store time-stamped data from RA modules forfast retrieval. In practice, one cache file may be written per mediaasset that is processed by the CI framework. Multiple data tracks mayexist within a media asset cache file for data from different RA modulesthat extracted information or properties from the media asset (with eachdata track addressed by a UID associated with an RA module). Each datatrack may include a list of time stamped samples of binary data of onetype (e.g., associated with one UID for an RA module), and several datatracks form one data cache file. A plugin may be code executable by amicroprocessor that includes one or more RA modules and one or more FAmodules that are used in combination to provide or implement a CIfeature.

FIG. 1 illustrates a computer system or network 100 as may be used by anoperator to process or analyze digital media assets. For example, thesystem 100 may include a data store or server 110 that stores and/orserves a plurality of media assets 114 over a network 120 such as theInternet, a local area network (LAN), a wide area network (WAN), or thelike to one or more media processing systems 130. For example, the mediaassets 114 may be temporal media such as video and/or audio files or maybe still images. In other embodiments, though, the data store 110 may belinked more directly via wired or wireless connections to the mediaprocessing system 130 and may in other cases be a part of thememory/data storage of system 130.

The media processing system 130 may be a workstation or other computingdevice that an media analyst or media asset user may operate to processmedia assets by running a set of CI tools or algorithms and/orapplications on the media assets. The system 130 may include amicroprocessor or CPU 132 that runs or manages input and output (I/O)devices 134, such as a keyboard, a mouse, a touch pad/screen, a printer,and the like to allow the media asset analyst/user to enter input ordata and to receive output such as via a graphical user interface (GUI)displayed on a screen of a monitor 136. For example, the system user mayoperate the I/O 134 to initiate the CI framework environment by causingthe system 130 via CPU 132 to run a CI framework 160 and may then enterinput or data to select one or more of the media assets 114 to processwith the CI framework 160 to create data cache files 172 and featurealgorithm tags 180 that are stored in memory 170.

The system 130 may also be used to run a media access or feedapplication 140 that may be used to create the GUI 138 to allow a userof system 130 to select assets 114 for processing and to feed/play theselected media 114 for analysis by the RA modules 164 (and/or later foraccessing by the FA modules 168). Further, the system 130 may includeone or more media processor applications 150 such as Adobe's PhotoShopthat may be used by an operator of the system when run by microprocessor132 to utilize and/or build upon the outputs of the feature algorithms168 (i.e., tags 180) and/or data generated by the RA modules 164 (e.g.,data tracks 176 or other metadata provided in memory/data cache 170 inthe data cache files 172 provided for each asset 114).

As shown, the microprocessor 132 runs or manages operation of executablecode in the form of a CI framework 160. The CI framework 160 includesone or more CI plugins 162 that each include executable code in the formof one or more raw analyzer (RA) modules 164 and one or more featurealgorithm (FA) modules 168. When used to process a media asset, the RAmodules 164 extract or generate information or properties regarding eachmedia asset 114 (such as a video file that is played in its entirety forat least partially concurrent analysis by all the RAs 164). The CIframework 160 is adapted to store in memory 170 a data cache file 172for each of the assets 114 that is processed by the CI framework 160.Each data cache file 172 is associated with a media asset file such aswith a link or asset ID 174, and each data cache file 172 also includesone or more data tracks 176 generated by one or more of the RA modules164. A UID 178 is used to link the data track 176 with the generating orproducing RA module 164 (which would also use this UID). Each data trackmay include a list of time stamped samples of binary data of aparticular type (such as histogram data for a video frame(s) or thelike). The FA modules 168 may act to process or use some or all the datatracks 176 so as to generate tags 180 that are linked by a UID 184 withthe producing or generating FA module 168.

In the system 130, there may be two application programming interfaces(APIs) for client to utilize (not shown in FIG. 1) such as components ofthe CI framework 160. First, a management API may be used to controlprimary ingestion of the media assets and related tasks. Second, afeature API may be used to allow/control access to the various featuresthat the CI framework 160 provides. The feature API may route throughthe access to the feature algorithm modules 168 from the analyzers 164or other plugins 162. The management API may provide a relativelyclassical set of functions that can be extended over time while in thecase of the feature API it may be unknown upfront which structure (orsignature) the individual feature algorithms 168 may have. Toaccommodate this arrangement, the CI framework 160 may use a client APIthat works more in the way of requests followed by responses rather thana large number of API calls that would have to be extended more often tocatch up with developments. The transmitted data in the framework 160(e.g., parameters and results) may have the form of eXtensible MarkupLanguage (XML) data, although some embodiments may use a binaryequivalent (e.g., class PropertyList which is a dynamical tree of simpletypes like numerical values and strings, or the like) of XML forincreased performance.

In a temporal architecture (i.e., one used more for analyzing video andaudio files), the CI framework 160 may basically work in two steps:ingestion and browsing. FIG. 2 illustrates operation of the system 100to provide media analysis 200 such as to analyze temporal media assets114. As shown, the analysis 200 starts at 204 such as with providing orloading the CI framework 160 onto a media processing system 130 andproviding the CI framework 160 with access to the data store 110 andmedia assets 114 (such as via network 120 or these assets may be localon system 130). At 210, the analysis 200 is shown to include loading theCI framework 160 and a user may provide input via GUI 138 or I/O 134 toselect CI plugins 162 for use in analyzing media data and/or byspecifically configuring a CI framework 160 with particular RA modules164 and FA modules 168. Other embodiments may allow all available RAmodules 164 and FA modules 168 to be run at 210 to create data cachefiles 172 and tags 180 for later selective use by media processorapplications 150.

At 220, the analysis 200 includes initiating media ingestion, and at 230a media asset/file 114 is selected and retrieved from store 110. At 240,the entire length of the media file 114 is played. In other words,during ingestion a media file 114 is played in its entire length pastall the raw analyzer modules 164 of all plugins 162 or active/selectedones of the plugins 162 in the CI framework 160. The CI framework 160may be triggered by an exporter plugin 140, such as Adobe's Premiere Proor similar products, that may act to hand single video frames to the CIframework 160.

At 250, each RA module 164 extracts properties or data from the mediaasset and at 260, this data is used to create a data cache file 172 foreach asset (identified with ID or link 174) to include a data track 176associated via UID 178 with a particular RA module 164. During thesesteps, each RA module 164 may extract immutable properties from themedia 114, which may be provided/played by application 140, such ashistogram data and stores it in the data cache 170. Such concurrentversus sequential analysis by the RA modules 164 of the media asset 114may be provided for performance reasons because some analyzers 164 maytake significant amounts of time per frame of media data. Hence, iftime-consuming RA modules 164 are used in a framework 160 it typicallyis better to run them only once in the background than during userinteraction (browsing). This may also apply to decoding of the media114.

An exemplary RA module 164 that may provide the functionality of steps250 and 260 is a face-recognition algorithm run by microprocessor 132,and most face-recognition modules may fall into this time-consumingcategory because they are more CPU-intensive than a simple histogramgenerator (e.g., another exemplary RA module 164). However, even thougha face-recognition-type RA module 164 provides high-level abstract data(e.g., face coordinates), from the CI framework 160 perspective, itproduces immutable data since no parameters are applied to yieldpotentially varying results. The same functionality (e.g., facedetection) may, however, be found again in one or more of the FA modules168 as well, e.g., for a closer look or analysis of the same or similardata extracted from a media asset as the FA modules 168 build upon anduse data 176 output from the RA modules 164.

The ingestion provided from steps 220 to 260 in method 200 of FIG. 2principally happens once for every new asset 114. In some cases, moredata may be added to an existing data cache file 172 from new rawanalyzers 164 later (e.g., a later version of an RA module 164 may beadded, a new RA module 164 may be added, and so on). Each RA module 164may be independent and not, typically, require existence of othermodules 164. In operation of system 100 and performance of method 200,an RA module 164 may receive audio or video frames in a certain format(or list thereof). The module 164 may also, in some cases, be able torequest to receive more than just one frame of video or audio at a time,e.g., for frame-to-frame motion analysis or the like. However, thesemultiple sets of frames include consecutive frames and their numbertypically will be limited. In some embodiments, algorithms or CI toolsthat operate with random access to frames are provided as FA modules 168rather than RA modules 164 such that they can access raw content datavia a special interface. It might be routed through the data cacheinterface and presented as a virtual data track, while it is not storedin the data cache file itself.

As part of storing at step 260, the CI framework 160 may include amechanism for storing data tracks 176 or RA module output only after itchecks whether the existing parameters are equal or different from thenewly generated parameters to determine whether the data cache file 172contains valid or obsolete data. In some embodiments of the system 100,a RA module 164 may be constructed once per asset run (step 240) andthen destroyed afterwards (in a step not shown in method 200 after step250). For example, a method (e.g., a DoOneFrame method) may be calledonly once for every frame of an asset 114 and the frame timestamps arecalled in the order of the timestamps in the asset 114 (with no frametypically being omitted). The frame format may remain the samethroughout the whole run in step 240.

Regarding the data case 170 of system 130, the data cache 170 stores thedata, such as RA output 178, passed to it for later fast browsing andretrieval in one binary cache file 172 per asset 114. Inside one cachefile 172, the data 176 may be organized by unique IDs (UIDs) 178 intomultiple logical data tracks. There may be one UID 178 per RA module 164and track 176, and the data may also be time stamped with the asset'smedia time stamps during ingestion 220. The time stamps for one UID 178steadily increase from one data sample to the next (which may be theonly condition for the time stamps). Data samples typically do not needto happen at any certain intervals nor do they typically need to be ofthe same size. One data sample may be thought to be the valid data fromthe time stamp on, which is attached to it, and the data sample lastsuntil the time stamp of the following data sample of the same track 176(or the end of the file 114).

Generally, the data cache 170 does not know anything about the meaningof the data stored in one sample or about the internal structure of thedata in files 172 or tags 180. The data cache may know UID, timestamp,and size in bytes of each data track sample 176 (or each file 172 or tag180). Interpretation of the data in the cache 170 generally is left upto CI plugins 162 that generate and use the data. The timestamps ofdifferent tracks generally do not have a certain relationship to eachother, but it may be useful to keep them relatively close so that when aportion of the cache file 172 is loaded into memory 170 all samples fromdifferent RA modules 164 for a certain period of time are present in thefile 172. The data cache 170 is named such from the fact that it can berecreated at any time from the original media assets 114.

With further regard to the RA modules 164 and data stored in the datacache, it may be useful to discuss interpreting binary data within themedia processing system 130. Since the CI framework 160 does not makeassumptions about the meaning or structure of the binary data which theRA modules 164 and FA modules 168 provide, some embodiments leave it tothe plugins 162 to handle the binary data directly while others placethis responsibility upon users of the FA modules 168. In other cases,though, a general data format that is flexible enough to incorporateevery structure that is deemed useful is utilized. For example, XML maybe utilized by the CI framework 160 for data exchange for both the RAmodules 164 and the FA modules 168, when data volumes are acceptablysized since it is a well-established format that many in the field knowhow to use and manipulate. In an XML-based data exchange embodiment, aclass PropertyList may be used to structure binary data into hierarchiesthat look similar to XML, while the data remains binary for higherperformance. In such an embodiment, FA modules 168 may use parametersthat are passed in from clients thus that dynamic adding of propertiesis permitted. Properties may be named such that users (e.g., clientapplications such as media processor applications 150) can recognizethem readily, which may be achieved via use of variably sized strings.In other embodiments, a proprietary binary format may be used for RA andFA module communications.

Returning again to FIG. 2 and the method 200, when ingestion is done,browsing may be initiated at 264. This may involve the second part of aplugin 162 being called, i.e., a feature algorithm (FA) module 168 maybe called. At 270, the method 200 looks to see if there are additionalFA modules 168 to be called. If so, step 276 involves calling the FAmodule 276 to use the RA module(s) 164 data in data cache fields 172 tocreate a new feature result that may be stored in memory as a tag 180linked to the FA module 168 by a UID 184. If there are no further FAmodules at 270, the method 200 may proceed at 280 with accessing the FAmodule output 180 with one or more client applications such as mediaprocessor applications 150. At 290, the method 200 is ended.

With further regard to the FA module 168, this algorithm may access thecache file 172 via a data API (not shown in FIG. 1) and retrieve thetemporal data useful for its particular operation from data tracks 176.The FA module 168 knows the meaning of the data for the UIDs 178 ituses, and it can also, in some cases, use UIDs 178 that were notgenerated by the RA modules 164 within the same plugin 162 but, instead,from any other CI plugin 162 in the framework 160. For example,histogram data of a certain type is only generated once by a particularRA module 164 for use by any FA module 168 rather than multiple timesfor each plugin 162 that may utilize such data for a media asset 114.The output of an FA module 168 may take a number of forms to practicethe system 100 and method 200. For example, each FA module 168 mayproduce lists of tags 180 (linked to the FA module 168 by FA UID 184),which may attribute a certain value or statement to a segment of theoriginal media asset 114, e.g., “Scene Number 1 ranges from 0.0 secondsto 3.12 seconds” for a scene determination algorithm. Tags 180 of thisform may be stored in the data cache 170 as shown in FIG. 1 and be usedby other FA modules 168 (e.g., the modules 168 may build on other FAmodules 168 and/or on RA module outputs in data cache files 172). The FAmodules 168 may also provide output that is stored differently such asXML, and/or in XMP and be used by FA modules 168 and/or by applications150 from this alternative/additional store in system 130. Either datastore technique provides much smaller volumes of data (or media assetmetadata) when compared with accessing the media asset itself.

The tags 180 may be immutable, e.g., shot or scene tags, and also highlyvolatile, e.g., the result of a visual search for objects in a video. Inthe first case (or immutable), there may be no parameter that is passedto the FA module 168 that may alter the result. However, in the secondor volatile case, a search or similar feature may pass parameters and areference image of an object to the FA module 168 to generate a rankedlist of similar objects. In such cases, the result or output of the FAmodule 168 may be consumed directly by an application 150 without itbeing stored as tag 180. In practice of system 100, different FA modules168 may need very different parameters ranging from none, over lists ofname value pairs, to binary image data, and the results/output may alsovary a large amount within a particular CI framework 160. Hence, in somecases, the parameter input and result output data of the FA modules 168may be described similarly to the other data stored in the data cache170 such as with a UID 184 along with a size for the binary blob of data(with, typically, a caller and a callee knowing the meaning of thisdata). Some embodiments may provide for more humanly readable formatssuch as XML/XMP while at the core FA modules 168 utilize and provide abinary data transfer in and out.

Another example of a CI plugin may be a scene-detection plugin 162 thatutilizes an RA module 164 to provide histograms 176 in a data cache file172 and uses an FA module 168 to generate (as shown at 180) a vector ofshot boundary timestamps. During the ingestion phase, the histogramsamples are generated by the RA module 164 from the video frames of anasset 114 played/provided by media access application 140, and thesesamples are stored for each frame as shown in at 176 with the RA's UID178 in a data cache file 172 for the identified asset 174. Then, thefeature algorithm 168 generates the vector of timestamps during one callbased on the stored histogram data (or RA output data) 176.

Regarding access to the original assets 114, some FA modules 168 may userandom access to the original footage or media data of an asset 114. Forinstance, a CPU-intensive algorithm/plugin 162 may have been run as rawanalyzer 164/feature algorithm 168 combination earlier for every fifthframe in an original video. However, for a certain workflow, it may beuseful to run it again for every frame in the vicinity of a particulartimestamp. In use of CI framework 160, a certain UID may be used to giveaccess to this data via an interface that provides the data to the RAmodules 164.

A plugin 162 may be implemented in system 130 as an executable piece ofcode (that may be dynamically linked) that brings a number of rawanalyzers 164 and feature algorithms 168. Plugins 162 may be classesthat implement a set of predefined methods, and the source code may beprovided as part of the CI framework module 160 run by microprocessor132. Where binaries are desirable that may not be part of the CIframework 160, these may be wrapped by classes defined by the set ofpredefined methods (which may be part of a plugin themselves such as animplementation of the abstract interface “ICITPlugin” or the like).Plugins 162 are useful (but not required) to bring a situation where itis reasonably easy to add new functionalities to a CI framework 160.

From the above discussion of FIGS. 1 and 2, it can be seen that the CIframework or toolkit concept provides an extensible framework forrunning content analysis on media assets. The CI framework providesoutput via the RA and FA modules to enhance understanding of an asset'scontent (e.g., discovery of visual or other properties in a video, astill image, and/or a sound recording such as by running algorithms thatact to detect faces, find logos, track objects, and the like). The CIframework or toolkit acts to facilitate collection of RA and FA modulesproviding content intelligence and then add and combine intelligence ontop of the RAs with FAs and on top of FAs with other FAs andapplications for ready integration by clients and client applications(e.g., the CI framework may be thought of as a bridge between CIproviders and client applications).

FIG. 3 illustrates another implementation of a CI framework 300 that maybe implemented via software modules or components run by one or moremicroprocessors of one or more computer devices (similar to that shownin FIG. 1). Note, in FIG. 3, the boxes labeled “AV” refer to audio/videodata, the boxes listed as “PL” refer to property lists (or thePropertyList class), dashed arrows are used to represent socket orshared memory connections, “param” refers to parameters, “RA” is used torefer to the raw analyzer phase, and “FA” is used to refer to thefeature algorithm phase of the CI framework 300.

The CI framework or toolkit 300 is described below by discussion of itsmain components. The CI framework 300 includes a library module 310(e.g., CIPApi.lib in FIG. 3), and client applications using theframework 300 may integrate the library module 310, which is adapted insome embodiments as a thin stub library that offers API calls in orderto communicate with the CIT service 320 (e.g., to trigger contentanalysis, retrieve results, and the like). In using a CITConnector API312, clients like application 304 may hand a file path to the CIframework 300. Other clients, like the media access application 314(which may be implemented as Adobe's Premiere Pro or similar products),that want to feed the CI framework 300 directly with AV data may useother connector APIs 318 (e.g., MediaSinkConnector API or the like).

The media access application 314 provides an exporter plugin 316 and animporter plugin 317. The application 314 may function to set up theexporter plugin 316 in order to communicate with the CI toolkit service320. The plugin 316 may retrieve single video frames and hand them overto the CI toolkit service 320 such as by leveraging the library module310. The importer plugin 317 may request results from the CI toolkitservice 320 later and drive further processing (e.g., XMP export,visualization, and the like).

The CI toolkit service 320 provides the main CI framework 300functionality and may run in a local service as shown in FIG. 3. Thisservice 320 hosts the CI framework or toolkit's core logic like amanager 322 for managing tasks, components for optional encoding anddecoding of media, a CI plugin framework 324 with CI plugins 326 thateach include one or more RA modules and one or more FA modules, a datacache 330 for storing data cache files 334 for each asset, andcomponents 336 exporting the analysis result data 338 as XMP, XML, orthe like. The CIT task manager 322 may function to manage requests tothe CI framework service 320, with the requests being run as separatethreads in some embodiments. The task manager 322 may be responsible forsetting up and managing concurrent tasks. CIT clients (such asapplications 304, 308) may get a handle to those tasks in order to pollfor progress and completion.

A media access module 340 may be provided in the CI framework service320. For clients 304, 308 that are not going to encode or decode thematerial to be analyzed on their own, the CI framework service 320 mayprovide a media access component 340 to support communications with theservice 320 and accessing of output data from plugins 326. The mediaaccess component 340 may leverage, for example, from existing frameworkssuch as Adobe's MediaCore, ffmpeg, or the like.

The plugin host or framework 324 includes RA/FA plugins 326 thatimplement one interface but are run in two phases: a raw analyzer phase(RA) and a feature algorithm phase (FA). During the RA phase, the pluginhost 324 fires raw data to the plugin 326 (e.g., RGB data of a videoframe), and the RA phase/module of the plugin 326 acts to computeintermediate results (e.g., histogram data). This result data is handedback to the plugin host 324 that, in turn, acts to save the data to anassociated data cache 330 in an asset-specific data cache file 334. Insome cases, these results 334 are considered immutable as they depend onparameters that change very rarely.

Later on, the FA module or phase of the plugin 326 may be triggered, andthe FA module may retrieve the RA result data from the data cache file334 of the data cache 330. The FA module of the plugin 326 then acts torun one or more algorithms on the data (e.g., to perform scene detectionin a video media asset). The result data from the FA phase/module may behanded back to the plugin host 324 in order to write it to XMP 338, tohand it to a client application 304, 308, and/or to store it within thedata cache 330. The FA module/phase results may be relatively volatileas they can depend upon direct user input/parameters. Note, plugins 326do not have to have both an RA and an FA module/phase and FA modules mayuse other plugin produced data (from an RA and/or an FA module) from thedata cache 330 and/or data cache file 334. One reason for splitting upthe plugins 326 into two phases (an RA and FA phase) is for improvedefficiency. The RA phase/modules may be thought of as doing the pre-workthat then can be used or built upon to make the FA phase/modules runmuch faster when the RA output data (in data cache file 334) isrequested.

In the framework 300, the data cache 330 may be used to persistentlystore RA and/or FA result data. The data cache 330 may be adapted toprovide an interface for quickly accessing the data produced for/from aparticular media asset. The RA modules and FA modules of the plugins 326may exchange data structures with the plugin host 324 that are notnecessarily known in advance. A node tree (e.g., a PropertyList (PL))may be provided for this purpose to allow handling of a tree structureof simple built-in types. In some embodiments of the framework 300, aquery API or similar module (not shown in FIG. 3) may be provided toretrieve results of an analysis performed by the plugins 326. Such aquery API may ask for high level data (e.g., number of faces in a videoframe) or low level data to run further analysis (e.g., histograminformation of certain video frames). The query API may also yieldcombined results from more than one plugin 326 or media asset at a time(e.g., object search over multiple assets). In some cases, results areretrieved by calling a particular FA module of a plugin 326 via its APIor the like.

As discussed above, each of the plugins 326 may be an executable pieceor module of code that brings any number of RAs and FAs together. Insome cases, a plugin 326 may provide code for one RA and one FA. Usingan extensible plugin architecture allows differing developers anddevelopment teams for content intelligence to provide new functionalityto the CI framework 300 without necessarily knowing about all theinternal code of the framework 300 and all aspects of media management(e.g., encoding/decoding). The developers/teams may simply use the CIhost 324 plugin API and can focus on the specific analysis orfunctionality they are interested in providing or achieving in the CIsystem/application.

The RA part or phase of each plugin 326 may function to extractsemi-immutable properties like histogram data from image or audio frames(e.g., from media data or media assets). The CI framework plugin host324 may fire raw data to the RA module, and the RA computes/creates someanalysis results. Output data is handed back to the plugin host 324 tostore into the data cache 330 in files 334. This phase of the plugin 326operation is called ingestion of a media asset. Both the RA and itsoutput data may be referred to by a single (the same) UID to link thesetogether. The RA may receive audio and image frames in a defined format(e.g., BGRA32 or the like) from the plugin host 324. It may also requestto receive more than just one frame at a time (e.g., a sliding window).In some cases, the ingestion step may happen once for every media assetbeing processed even though more data may be added to an existing datacache file 334 by additional RAs at a later point in time or lateroperating step of CI framework 300.

The FA modules or portions of the plugins 326 may act to retrieve theRA(s) result data from the data cache 330 and turn it into its ownoutput. For example, each FA module may turn the RA data into lists oftags (e.g., specific content-related metadata like “Scene Number 1ranges from 0.0 seconds to 3.12 seconds” or the like). This phase ofplugin operations may be called browsing. The FA module and the type oftag may also be referred to by one (the same) UID. The result tags or FAresults of an FA module may be used by other FA modules. The tags orresults may be stored as XMP, in XLM, or the like and be used from thistype of storage by client applications. Tags may be immutable(shot/scene tags or the like) or be volatile (the result of visualsearch for objects in a video, for example). A search might passparameters and a reference image of an object to the FA module togenerate a ranked list of similar objects. In such a case, the resultmight be consumed by the application directly with or without storing itin memory. The FA module knows the meaning of the RA output data that ituses, and the FA can be dependent on several RA modules and theiroutputs (which may be part of the same or a different plugin 326). Thistype of data dependency 400 is shown in FIG. 4. As shown, RA modules 410generate data tracks 420 that are linked to the generating/producing RAmodule 410 by a UID. The set of FA modules 430 may utilize data from oneor more of the RA modules 410, and these RA modules 410 may be from thesame or differing one of the plugins (e.g., FA3 uses data from RA1 andalso RA n while FA1 is dependent on data from RA1 and also from RA2).

One example for a plugin 326 of FIG. 3 is a scene detection plugin thatacts to determine data useful for splitting a video into its differentscenes. In this case, the RA module may consume each video frame andcompute histogram data for the frame. Later on, the related FA module ofthe plugin (or another plugin) may determine scenes dependent on certainchanges to the histograms from frame to frame (e.g., the FA moduleresults are dependent on the RA results/histograms stored in a datatrack in the data cache file 334 in the data cache, which is accessed bythe FA module). Again, a reason for splitting up the plugins 326 intoRAs and FAs is efficiency. The RAs do the pre-work, which then makes theFAs run much faster when they access the data they use to produce aparticular output or feature (which may, in turn, be used as input to aclient application 304, 308). The split may be largely logical, withonly few consequences in the code structure. The code for the RAs andthe FAs may live or reside in the same plugin class (or binary, onceapplicable) and may share common functionality.

During plugin discovery, a plugin API for the host 324 or other portionof service 320 may be used to retrieve the MIN for the RA modules. Inanother step, the CI plugin host 324 of service 320 may act to sort outthe dependencies (e.g., while the RA modules are independent from eachother, the FA modules may depend upon the output data of other plugins326 and their RA modules and/or FA modules). FIG. 5 illustrates a graph500 illustrating use of a CI framework such as framework 300 to analyzemedia assets and showing how a new feature can depend upon new andexisting features and raw data alike. As shown, a media asset 510 in theform of an input video file may be played for a set of analyzer plugins514 with a set of raw analyzers RA1 to RA4, which act duringdecode/ingestion 520 to analyze the asset.

During ingestion, the RAs of plugins 514 act to create output or datatracks 530 that are associated to the RAs by UIDs and that are stored ina data cache file in data cache 534. In the data cache 534, the data 530may be organized by UIDs into multiple logical data tracks. There may beone UID per raw analyzer and track. The data may be timestampedcorresponding to the asset media timestamps received during ingestion520, with the timestamps for one UID increasing from one data sample tothe next. Data samples do not need to occur at certain intervals, andthey typically do not need to be the same size. A data sample may beconsidered valid until the next data sample (or end of the asset 510).

During browsing 550, a data API 540 may be used to allow a number of FAmodules provided by plugins 514 to access the data tracks (or output ofRAs) 530, and again, the dependency is not necessarily a one-to-onedependency or limited just to an RA in the same plugin 514. Further, anFA 556 may also be dependent on output of another FA 556 (e.g., FIG. 5shows that FA2 and FA3 are dependent upon or use the output results ofFA1). The CI framework API 558 provides data access to clientapplications 560 such that these client applications 558 may utilize theFA outputs (e.g., tags or lists of tags).

Although the invention has been described and illustrated with a certaindegree of particularity, it is understood that the present disclosurehas been made only by way of example and that numerous changes in thecombination and arrangement of parts can be resorted to by those skilledin the art without departing from the spirit and scope of the invention,as hereinafter claimed. For example, a CI framework may also be run in aserver environment rather than the client/user computer as shown in FIG.1.

Additionally, it may be useful at this point to provide further overviewof the CI toolkit or framework (such as may be implemented as shown withCI framework 160 in FIG. 1). The CI framework may be adapted to work ona desktop device such as with conventional operating systems (e.g.,Windows or Mac operating systems) and/or adapted for use on a server(e.g., a Linux-based server). This may mean that the CI frameworkdepends on different components, e.g., for media access, on thedifferent platforms. For this reason, the CI framework may usecomponents such as Adobe's MediaCore or the like in some implementationsrather than being implemented as a part of such components. The CIframework may be accessed from different applications on a desktop orserver, and there may be heavy parallel background tasks being performed(e.g., the initial indexing/ingestion of the media assets), which may berun on assets that are too large to be copied efficiently so that theyare shared by different processes. In some cases, a central instance permachine is provided of a CI framework that manages requests fromdifferent processes (in some cases, even if the individual tasksperformed are rather simple but CPU intensive).

Because of these issues, the CI framework may be a process that runslike a local service and spawns child threads and processes asappropriate. The CI framework process may, however, be headless (e.g.,there can be a controlling GUI for convenience but not required). The CIframework process may receive commands via socket communication fromeither local or distributed clients. For clients (e.g., pointapplications), this communication may be hidden behind client libraries,which the clients can link to and use as if they were simple in-processlibraries. For example, a C++ library (e.g., one that implements a classConnector or the like) may be used that clients may use to access alocal CI framework and/or a remote CI framework. The local CI frameworkmay also be accessed in-process, with or without multithreading (e.g.,for debugging purposes). Out-of-process access may work via sockets soas to allow access to a local or remote CI framework.

Regarding a still image architecture for the CI frameworks, the abovedescription stresses use of the CI framework for processing temporalmedia data such as video or audio files. However, the concepts andfunctions described may also be used for processing media assets ormedia data for still images. Still images lack the tight temporalrelationship of video frames in a video. Consequently, algorithms thatmake use of this relationship may not be used in a CI frameworkprocessing still image assets. However, the split of algorithms into rawanalyzers and features is useful for still image processing since,again, a first ingestion step may be used to generate raw metadata thatcan then be used by the feature algorithms, e.g., to later search withinan image for an object or face. A data cache may again be utilized, butit may be modified such that many assets (and/or their raw metadata) arestored in one cache file such that many images may be searched orotherwise processed by a feature algorithm (rather than each assethaving a data cache file). File paths may be utilized in place oftimestamps for addressing images, but timestamps or other informationmay be used to provide an index value in an array for referencing animage file in the data cache.

It may be useful to provide another specific working example of acontent stream being analyzed by two or more RAs and then the outputbeing used by an FA or two (e.g., with one FA using the results ofanother FA or the like). For a feature “scene similarity” for example,RA1 may compute histogram data per frame and store these results in thedata cache in a data track while RA2 may compute color swatches andstores them as well in the data cache file for the same media asset. FA1may then calculate the scene-cuts in the video (or other asset) from thedata cache file values stored by RA1 and/or RA2 and afterwards FA2 maycalculate an average color swatch per scene depending on the output ofFA1. Then, further, FA3 may take pairs of average color swatches fromFA2 and compares them, which may result in an N by N matrix ofsimilarity values for the N scenes.

It may also be useful at this point to provide an example of the form anasset file may take in a data cache, e.g., explaining how the datadiffers from the original file. Building on the relatively simpleexamples above, the per-frame histogram data may be a three-dimensionalarray with 16 possible values for each dimension (e.g., the three colorcomponents red, green, and blue) resulting in a total of 4096 buckets.Each entry in the array is an integer value of how many pixels in oneframe have a color-tupel which falls into this “bucket”. If each integervalue is 4 bytes large, that results in 16384 bytes per frame. ADV-encoded video frame has around 145000 bytes of data, for comparison,and the histogram data is not required for every frame, but everyfourth, in the current implementation. Since many of the buckets areusually empty, a simple RLE data compression would achieve furtherreduction. The color swatches are a lot smaller still as they are a listof five RGB-color triples. Each RGB color is made up of three 8-bytedouble values, which results in a total of 120 bytes per frame.

Regarding improvements in efficiency provided by a CI frameworkimplementation, it may be said that the performance improvements can behuge in one case while being relatively small but still significant inanother, depending on the algorithm. For example, for N raw-analyzersthat each access the raw pixels of a video, the speedup is around afactor N. This is because the decoding of the video only needs to happenonce for all RAs, and there are many algorithms that are computationallycheap, so that the frame-serving is indeed the bottleneck. But, forinstance for the scene-detection, the gain is huge (e.g., greater than1000 times or the like) when a user plays with the threshold and wantsto see what different results he gets for each value. In that use-case,the result available is mostly instantaneous, whereas without the datacache the video would be decoded multiple times.

Typically, the CI framework or toolkit interface to the outside world israther small. For example, a call to Connector::Open( )loads the CIT.dlland enumerates the available plug-ins/algorithms.Ingest(analyzerUidList, videoFile) may be used to run all specifiedanalyzers on a video file (“videoFile”). One or multiple calls toCallFeatureAlgorithm(FA-ULD, parameters, output, datacacheFile) may beused to run the feature algorithms. As a final/next step, the user maywalk the resulting propertyList “output” to retrieve the individualresults, which is very similar to XML-parsing (in fact the propertyListcan be converted into XML, but that is not the most attractiveproposition for a developer using the CIT interfaces in C++).

The CI framework or toolkit approach is believed to be very useful inthe realm of content-intelligence. Typical artificialintelligence/content intelligence (AI/CI) related tasks are very contextsensitive, which for instance means that one parameter “threshold”,which works well for one situation is useless in another. Making analgorithm robust may cause a developer to either adjust the parameter toeach situation or leave the parameter alone and interpret the resultsaccordingly. However, the second algorithm which does that might not berelated at all to the original algorithm. For instance, the facedetector may tend to produce false positives, i.e. it sees faces wherethere are none. To address this problem, the faces in one scene may begrouped (e.g., reusing scene detection along the way) and tracking howeach face moves through a scene along a path. The false positivesuppression is not the main output of the algorithm, but it is one ofthe reasons why it is deployed. Of course, this functionality may be allput into one algorithm, but that would suppress collaboration in theteam and probably result in the usual convoluted mess in the code, whichmay make it difficult to take the code apart later on and improve it. Incontrast, the CI framework described herein provides plug-ability ofrelatively simple algorithms, which significantly enhances the abilityto develop more abstract ones or feature algorithms and the like.

Embodiments of the subject matter described in this specification can beimplemented as one or more computer program products, i.e., one or moremodules of computer program instructions encoded on a computer-readablemedium for execution by, or to control the operation of, data processingapparatus. For example, the modules used to provide the CI framework 160such as the RA modules 164, the FA modules 168, and the like may beprovided in such computer-readable medium and executed by a processor orthe like. The computer-readable medium can be a machine-readable storagedevice, a machine-readable storage substrate, a memory device, acomposition of matter affecting a machine-readable propagated signal, ora combination of one or more of them. The term “form generating system”encompasses all apparatus, devices, and machines for processing data,including by way of example a programmable processor, a computer, ormultiple processors or computers. The system (such as systems 100 and130 of FIG. 1) can include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form; including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub-programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read-only memory or a random access memoryor both. Generally, the elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. The techniques described herein may beimplemented by a computer system configured to provide the functionalitydescribed.

For example, FIG. 1 is a block diagram illustrating one embodiment of acomputer system 100 and media processing system 130 configured toimplement the methods described herein. In different embodiments,computer systems 110 and 130 may be any of various types of devices,including, but not limited to a personal computer system, desktopcomputer, laptop, notebook, or netbook computer, mainframe computersystem, handheld computer, workstation, network computer, applicationserver, storage device, a consumer electronics device such as a camera,camcorder, set top box, mobile device, video game console, handheldvideo game device, a peripheral device such as a switch, modem, router,or, in general, any type of computing or electronic device.

Typically, a computer will also include, or be operatively coupled toreceive data from or transfer data to, or both, one or more mass storagedevices for storing data, e.g., magnetic, magneto-optical disks, oroptical disks. However, a computer need not have such devices. Moreover,a computer can be embedded in another device, e.g., a mobile telephone,a personal digital assistant (PDA), a mobile audio player, a GlobalPositioning System (GPS) receiver, a digital camera, to name just a few.Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user (with an I/O portion 134 ofsystem 130 or monitor 136 of system 130 or the like), embodiments of thesubject matter described in this specification can be implemented on acomputer having a display device, e.g., a CRT (cathode ray tube) or LCD(liquid crystal display) monitor, for displaying information to the userand a keyboard and a pointing device, e.g., a mouse or a trackball, bywhich the user can provide input to the computer. Other kinds of devicescan be used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what may beclaimed, but rather as descriptions of features specific to particularembodiments of the invention. Certain features that are described inthis specification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and/or parallelprocessing may be advantageous. Moreover, the separation of varioussystem components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software and/orhardware product or packaged into multiple software and/or hardwareproducts.

At this point, it may be useful to provide another implementation or useexample for the CI framework or CI toolkit (CIT) that highlightsexemplary virtues or advantages of the CIT. The CIT may be useful to inperforming video summarization such as in a system that rendersrepresentative short summaries of a video, like animated chapterthumbnails (e.g., for DVD authoring or other tasks). In thisimplementation, a first step may involve segmenting the given inputvideo into smaller chunks using a shot detection algorithm, with an RAto calculate histograms for individual frames and an FA to determineshot boundaries based on the histogram values, for example. The videosummaries may now be generated using the CIT based on a certain set ofcriteria. For example, the criteria may include no black frames, noframes with titles/credits (using OCR, color analysis, and/or the like),shots with high activity (using an Activity Estimator), shots with oneor more people (using Face Detection), do not split dialogues whenselecting content for the summaries (using Speech-to-Text and AudioSilence Detection), and so on.

All the algorithms defining the criteria may be implemented as CIT RA/FAplug-in combinations. The CIT allows one to structure the code intoplug-ins, and a new and an older version of the algorithm may beavailable simultaneously without any copying of code or binaries,because the new algorithm will differ only in those parts which areactually new (e.g., a new top level plug-in when thinking in thefunctionality-tree which is built up by plug-ins using each other'soutput). As a result, updates to existing algorithms may be achieved,for example, by simply updating the corresponding CIT plug-in.Additionally, new criteria (e.g. a Sound Classifier) can be added bysimply adding a new CIT plug-in. So, not only can the new functionalitybe safely built from the old code, but it can also be safely developedand tested because the old algorithm will always be available in thevery same source code for immediate comparison. This example illustratesthe power of the CIT. In contrast, prior CI systems typically focus ontailored solutions for specific use cases (e.g., face recognition), butthey did not care about making it easy combining results of differenttypes of content intelligence. CIT closes this gap in providing a systemfor content intelligence lego, which means to combine a set of isolatedalgorithms to retrieve complex information about content.

We claim:
 1. A method, comprising: providing sequential access to aplurality of portions of a stored digital media asset; receiving aselection of raw analyzer modules, each of the respective raw analyzermodules to produce a respective set of raw analyzer result data, each ofthe selection of the raw analyzer modules accepts no input parametersand is executed exactly once during the playing back of the respectiveportions of the stored digital media asset; using one or moreprocessors, accessing the stored digital media asset and playing backthe respective portions of the stored digital media asset whileconcurrently executing the selection of the raw analyzer modules toanalyze the portions of the digital media asset to produce therespective sets of raw analyzer result data; storing, in a data cachethe sets of raw analyzer result data; and executing a feature algorithmmodule to access at least one of the sets of raw analyzer result data inthe data cache to generate an analysis result for the digital mediaasset based on the at least one of the sets of raw analyzer result dataaccessed by the feature algorithm module, the feature algorithm moduleaccepts at least one input parameter and is executed more than oncebased on more than one value for the at least one input parameter. 2.The method of claim 1, wherein the digital media asset comprises anaudio or video file and the portions comprise temporally-related framesof the audio or video file that include timestamps.
 3. The method ofclaim 2, wherein each of the sets of the raw analyzer result datacomprises a data track providing metadata for the frames extracted by anassociated one of the raw analyzer modules and associated with theframes of the digital media asset via the timestamps.
 4. The method ofclaim 3, wherein the metadata comprises an immutable property of contentin the audio or video file.
 5. The method of claim 1, further comprisingproviding the feature algorithm module and at least one of the selectionof raw analyzer modules in a plugin.
 6. The method of claim 1, furthercomprising running an additional feature algorithm module to generate anadditional analysis result based on the analysis result of the featurealgorithm module and based on at least one of the sets of raw analyzerresult data in the data cache.
 7. A non-transitory computer-readablestorage medium with an executable program stored thereon, wherein theprogram instructs one or more computers to perform operationscomprising: receiving a selection identifying a portion of a pluralityof raw analyzer modules, each of the respective raw analyzer modules toproduce a respective set of raw analyzer result data, the plurality ofraw analyzer modules accepts no input parameters and is executed exactlyonce during the playing back of the respective portions of the storeddigital media asset; playing a media file including time-related frames,the media file retrieved from a memory; during the playing, extracting aplurality of sets of metadata using the identified portion of theplurality of raw analyzer modules, wherein at least a portion of the rawanalyzer modules operate concurrently; storing the sets of metadata;accessing at least one of the sets of metadata; and based on theaccessing, generating a secondary analysis result for the media fileusing a feature module algorithm module, the feature algorithm moduleaccepts at least one input parameter and is executed more than oncebased on more than one value for the at least one input parameter. 8.The non-transitory computer-readable storage medium of claim 7, whereinat least a portion of the sets of the metadata comprise semi-immutableproperties determined for the time-related frames.
 9. The non-transitorycomputer-readable storage medium of claim 8, wherein the immutableproperties comprise histograms for the time-related frames.
 10. Thenon-transitory computer-readable storage medium of claim 7, wherein theaccessing further comprises using one or more time stamps associatedwith the time-related frames to access metadata within the at least oneof the sets of metadata.
 11. The non-transitory computer-readablestorage medium of claim 7, wherein the metadata is based on thetime-related frames.
 12. The non-transitory computer-readable storagemedium of claim 7, wherein the secondary analysis result comprisesidentification of an object in one or more of the time-related framesbased on processing of at least one of the sets of the metadata.
 13. Acomputer system comprising: a media import module to play a digitalmedia asset and provide concurrent access to the digital media asset byraw analyzer modules selected by a user, each of the selection of theraw analyzer modules accepts no input parameters and is executed exactlyonce during the playing back of the respective portions of the storeddigital media asset; a plurality of media asset analysis plugins togenerate analysis results, each of the media asset analysis pluginscomprising: at least one raw analyzer module, each of the at least oneraw analyzer module to create, using one or more processors, a datatrack associated with at least one portion of the digital media assetplayed by the media import module to store the data track, and at leastone feature algorithm module to access each of one or more data tracksto generate a feature analysis result based on the one or more datatracks, the feature algorithm module accepts at least one inputparameter and is executed more than once based on more than one valuefor the at least one input parameter.
 14. The system of claim 13,wherein the digital media asset comprises an audio or video file withtime-related frames and wherein the one or more data tracks includeproperties extracted from the frames and associated with the frames viatimestamp data from the audio or video file.
 15. The system of claim 14,wherein the at least one feature algorithm module accesses data in theone or more data tracks by providing a portion of the timestamp data.16. The system of claim 14, wherein the data in the one or more datatracks comprises a semi-immutable property of content of one or more ofthe frames.
 17. The system of claim 13, wherein the media import moduleplays an entire length of the digital media asset.
 18. The system ofclaim 13, further comprising a client application receiving the featureanalysis result as input and wherein the feature algorithm moduleaccesses at least one data track associated with at least one rawanalyzer module provided in a different one of the media asset analysisplugins.
 19. The method of claim 1, further comprising: storing in thedata cache the analysis result, wherein the analysis result is linked tothe feature algorithm module by a second unique identifier for thefeature algorithm module; and accessing, by an application, the analysisresult using the second unique identifier.
 20. The method of claim 1,wherein the analysis result comprises a tag indicating a time range ofat least one segment of the digital media asset and describing acharacteristic of the at least one segment.